Search for: All records

Abstract Large-scale processing and dissemination of distributed acoustic sensing (DAS) data are among the greatest computational challenges and opportunities of seismological research today. Current data formats and computing infrastructure are not well-adapted or user-friendly for large-scale processing. We propose an innovative, cloud-native solution for DAS seismology using the MinIO open-source object storage framework. We develop data schema for cloud-optimized data formats—Zarr and TileDB, which we deploy on a local object storage service compatible with the Amazon Web Services (AWS) storage system. We benchmark reading and writing performance for various data schema using canonical use cases in seismology. We test our framework on a local server and AWS. We find much-improved performance in compute time and memory throughout when using TileDB and Zarr compared to the conventional HDF5 data format. We demonstrate the platform with a computing heavy use case in seismology: ambient noise seismology of DAS data. We process one month of data, pairing all 2089 channels within 24 hr using AWS Batch autoscaling. 

The field of oceanography is transitioning from data-poor to data-rich, thanks in part to increased deployment ofin-situplatforms and sensors, such as those that instrument the US-funded Ocean Observatories Initiative (OOI). However, generating science-ready data products from these sensors, particularly those making biogeochemical measurements, often requires extensive end-user calibration and validation procedures, which can present a significant barrier. Openly available community-developed and -vetted Best Practices contribute to overcoming such barriers, but collaboratively developing user-friendly Best Practices can be challenging. Here we describe the process undertaken by the NSF-funded OOI Biogeochemical Sensor Data Working Group to develop Best Practices for creating science-ready biogeochemical data products from OOI data, culminating in the publication of the GOOS-endorsed OOI Biogeochemical Sensor Data Best Practices and User Guide. For Best Practices related to ocean observatories, engaging observatory staff is crucial, but having a “user-defined” process ensures the final product addresses user needs. Our process prioritized bringing together a diverse team and creating an inclusive environment where all participants could effectively contribute. Incorporating the perspectives of a wide range of experts and prospective end users through an iterative review process that included “Beta Testers’’ enabled us to produce a final product that combines technical information with a user-friendly structure that illustrates data analysis pipelines via flowcharts and worked examples accompanied by pseudo-code. Our process and its impact on improving the accessibility and utility of the end product provides a roadmap for other groups undertaking similar community-driven activities to develop and disseminate new Ocean Best Practices.